A striking observation about iterative magnitude pruning (IMP; Frankle et al. 2020) is that--after just a few hundred steps of dense training--the method can find a sparse sub-network that can be trained to the same accuracy as the dense network. However, the same does not hold at step 0, i.e. random initialization. In this work, we seek to understand how this early phase of pre-training leads to a good initialization for IMP both through the lens of the data distribution and the loss landscape geometry. Empirically we observe that, holding the number of pre-training iterations constant, training on a small fraction of (randomly chosen) data suffices to obtain an equally good initialization for IMP. We additionally observe that by pre-training only on easy training data, we can decrease the number of steps necessary to find a good initialization for IMP compared to training on the full dataset or a randomly chosen subset. Finally, we identify novel properties of the loss landscape of dense networks that are predictive of IMP performance, showing in particular that more examples being linearly mode connected in the dense network correlates well with good initializations for IMP. Combined, these results provide new insight into the role played by the early phase training in IMP.

Density estimation is one of the fundamental problems in statistics.

Due to the non-convex nature of such joint estimators, the theoretical justification of their efficiency is typically challenging.

Due to the non-convex nature of such joint estimators, the theoretical justification of their efficiency is typically challenging.

Suppose that we observe i.i.d.

We thank the reviewers for their comments. All reviewers agree that our theoretical results are solid and well-explained. "performing a clustering on the initial models should already give the right clusters" . Experiments We observe that the following common questions about experiments were raised by the reviewers. "W as a separate dataset for parameter evaluation used?": We choose the hyperparameters within a wide range.

Suppose that we observe i.i.d.

A striking observation about iterative magnitude pruning (IMP; Frankle et al. 2020) is that--after just a few hundred steps of dense training--the method can find a sparse sub-network that can be trained to the same accuracy as the dense network. However, the same does not hold at step 0, i.e. random initialization. In this work, we seek to understand how this early phase of pre-training leads to a good initialization for IMP both through the lens of the data distribution and the loss landscape geometry. Empirically we observe that, holding the number of pre-training iterations constant, training on a small fraction of (randomly chosen) data suffices to obtain an equally good initialization for IMP. We additionally observe that by pre-training only on "easy" training data, we can decrease the number of steps necessary to find a good initialization for IMP compared to training on the full dataset or a randomly chosen subset.

Given iid observations from an unknown absolute continuous distribution defined on some domain Ω, we propose a nonparametric method to learn a piecewise constant function to approximate the underlying probability density function. Our density estimate is a piecewise constant function defined on a binary partition of Ω. The key ingredient of the algorithm is to use discrepancy, a concept originates from Quasi Monte Carlo analysis, to control the partition process. The resulting algorithm is simple, efficient, and has a provable convergence rate. We empirically demonstrate its efficiency as a density estimation method. We also show how it can be utilized to find good initializations for k-means.